Logics and practices of transparency and opacity in real-world applications of public sector machine learning

Machine learning systems are increasingly used to support public sector decision-making across a variety of sectors. Given concerns around accountability in these domains, and amidst accusations of intentional or unintentional bias, there have been increased calls for transparency of these technologies. Few, however, have considered how logics and practices concerning transparency have been understood by those involved in the machine learning systems already being piloted and deployed in public bodies today. This short paper distils insights about transparency on the ground from interviews with 27 such actors, largely public servants and relevant contractors, across 5 OECD countries. Considering transparency and opacity in relation to trust and buy-in, better decision-making, and the avoidance of gaming, it seeks to provide useful insights for those hoping to develop socio-technical approaches to transparency that might be useful to practitioners on-the-ground. An extended, archival version of this paper is available as Veale M., Van Kleek M., & Binns R. (2018). `Fairness and accountability design needs for algorithmic support in high-stakes public sector decision-making' Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI'18), http://doi.org/10.1145/3173574.3174014.Comment: 5 pages, 0 figures, presented as a talk at the 2017 Workshop on Fairness, Accountability, and Transparency in Machine Learning (FAT/ML 2017), Halifax, Canada, August 14, 201

Eavesdropping Whilst You're Shopping: Balancing Personalisation and Privacy in Connected Retail Spaces

Physical retailers, who once led the way in tracking with loyalty cards and `reverse appends', now lag behind online competitors. Yet we might be seeing these tables turn, as many increasingly deploy technologies ranging from simple sensors to advanced emotion detection systems, even enabling them to tailor prices and shopping experiences on a per-customer basis. Here, we examine these in-store tracking technologies in the retail context, and evaluate them from both technical and regulatory standpoints. We first introduce the relevant technologies in context, before considering privacy impacts, the current remedies individuals might seek through technology and the law, and those remedies' limitations. To illustrate challenging tensions in this space we consider the feasibility of technical and legal approaches to both a) the recent `Go' store concept from Amazon which requires fine-grained, multi-modal tracking to function as a shop, and b) current challenges in opting in or out of increasingly pervasive passive Wi-Fi tracking. The `Go' store presents significant challenges with its legality in Europe significantly unclear and unilateral, technical measures to avoid biometric tracking likely ineffective. In the case of MAC addresses, we see a difficult-to-reconcile clash between privacy-as-confidentiality and privacy-as-control, and suggest a technical framework which might help balance the two. Significant challenges exist when seeking to balance personalisation with privacy, and researchers must work together, including across the boundaries of preferred privacy definitions, to come up with solutions that draw on both technology and the legal frameworks to provide effective and proportionate protection. Retailers, simultaneously, must ensure that their tracking is not just legal, but worthy of the trust of concerned data subjects.Comment: 10 pages, 1 figure, Proceedings of the PETRAS/IoTUK/IET Living in the Internet of Things Conference, London, United Kingdom, 28-29 March 201

Slave to the Algorithm? Why a \u27Right to an Explanation\u27 Is Probably Not the Remedy You Are Looking For

Algorithms, particularly machine learning (ML) algorithms, are increasingly important to individuals’ lives, but have caused a range of concerns revolving mainly around unfairness, discrimination and opacity. Transparency in the form of a “right to an explanation” has emerged as a compellingly attractive remedy since it intuitively promises to open the algorithmic “black box” to promote challenge, redress, and hopefully heightened accountability. Amidst the general furore over algorithmic bias we describe, any remedy in a storm has looked attractive. However, we argue that a right to an explanation in the EU General Data Protection Regulation (GDPR) is unlikely to present a complete remedy to algorithmic harms, particularly in some of the core “algorithmic war stories” that have shaped recent attitudes in this domain. Firstly, the law is restrictive, unclear, or even paradoxical concerning when any explanation-related right can be triggered. Secondly, even navigating this, the legal conception of explanations as “meaningful information about the logic of processing” may not be provided by the kind of ML “explanations” computer scientists have developed, partially in response. ML explanations are restricted both by the type of explanation sought, the dimensionality of the domain and the type of user seeking an explanation. However, “subject-centric explanations (SCEs) focussing on particular regions of a model around a query show promise for interactive exploration, as do explanation systems based on learning a model from outside rather than taking it apart (pedagogical versus decompositional explanations) in dodging developers\u27 worries of intellectual property or trade secrets disclosure. Based on our analysis, we fear that the search for a “right to an explanation” in the GDPR may be at best distracting, and at worst nurture a new kind of “transparency fallacy.” But all is not lost. We argue that other parts of the GDPR related (i) to the right to erasure ( right to be forgotten ) and the right to data portability; and (ii) to privacy by design, Data Protection Impact Assessments and certification and privacy seals, may have the seeds we can use to make algorithms more responsible, explicable, and human-centered

Algorithms that Remember: Model Inversion Attacks and Data Protection Law

Many individuals are concerned about the governance of machine learning systems and the prevention of algorithmic harms. The EU's recent General Data Protection Regulation (GDPR) has been seen as a core tool for achieving better governance of this area. While the GDPR does apply to the use of models in some limited situations, most of its provisions relate to the governance of personal data, while models have traditionally been seen as intellectual property. We present recent work from the information security literature around `model inversion' and `membership inference' attacks, which indicate that the process of turning training data into machine learned systems is not one-way, and demonstrate how this could lead some models to be legally classified as personal data. Taking this as a probing experiment, we explore the different rights and obligations this would trigger and their utility, and posit future directions for algorithmic governance and regulation.Comment: 15 pages, 1 figur

Book review: coffee by Gavin Fridell

Coffee is one of the most valuable Southern exports, generating billions of dollars in corporate profits each year, yet the majority of the world’s 25 million coffee families live in relative poverty. Gavin Fridell’s book aims to analyse the key factors shaping the coffee business. Michael Veale is impressed and recommends this to readers interested in trade, certification, and exploitation

Book review: fantasy islands: Chinese dreams and ecological fears in an age of climate crisis by Julie Sze

Fantasy Islands aims to explore Chinese, European, and American eco-desire and eco-technological dreams, and examines the solutions they offer to environmental degradation in this age of global economic change. Michael Veale finds this a refreshing read

Book review: the problem-solving capacity of the modern state

The early 21st century has presented considerable challenges to the problem-solving capacity of the contemporary state in the industrialised world. Among the many uncertainties, anxieties and tensions, it is, however, the cumulative challenge of fiscal austerity, demographic developments, and climate change that presents the key test for contemporary states. This book considers the state of governance in the current period of profound uncertainty and put forwards a concrete set of proposals for the way ahead. Michael Veale recommends this book to students and scholars whom are interested in what the regulation literature has to say about complex problems

Book review: action research for sustainability by Jonas Egmose

Sustainability researchers often forget that many of their fellow humans, even if thoroughly convinced of the problems of environmental degradation, have differing pressing interests and concerns that leave limited time or energy to engage with grand societal challenges, writes Michael Veale. Action Research for Sustainability is interesting reading for researchers studying social systems, for those designing social science education, and a loud call for action for getting up from desks and going into relevant communities – at least occasionally

Book review: the formula: how algorithms solve all our problems … and create more by Luke Dormehl

Amidst the confusion and the hype around big data, Luke Dormehl’s The Formula charts the growing influence of algorithms in modern society and how they affect our daily lives. Michael Veale finds that this book’s strength lies in providing background on the personalities behind the hype. He recommends it to people new to the field looking for a basic recent history, and to those looking for an understanding of the mentalities that both seek to fit formulas to the world, and fit the world into formulas

Book review: Service automation: robots and the future ofwork by Leslie P. Willcocks and Mary C. Lacity

The topic of service automation is typically shrouded in both hype and fear. In Service Automation: Robots and the Future of Work, Leslie P. Willcocks and Mary C. Lacity present the results of a survey, client case studies and interviews with service automation clients, providers and advisers, engaging with both the benefits and possible challenges of present and future service automation technologies. While the authors could have been bolder in exploring some of the issues raised, Michael Veale recommends this book as a useful starting point for those looking to understand the growing implementation of automation in a variety of operational contexts